Add a tutorial that runs GPU training jobs on a Spot GPU node pool - #52

Motivation

The repository has no GPU example.
On AKS, GPU workloads run on node pools of NVIDIA GPU VM sizes, and a Spot pool that scales from zero is the cheap way to run them, at the price of evictions.
LocalStack for Azure now emulates that flow: GPU nodes carry the labels and taints AKS gives them and advertise the GPUs of their size, the cluster autoscaler grows a pool from zero for a pod that asks for a GPU, and az vmss simulate-eviction evicts a Spot node as AKS does (see Run GPU workloads).
This tutorial walks through that flow with training jobs that KEDA starts for each message on a Service Bus queue, and runs unchanged on Azure, where the job trains on the GPU, and on the emulator, where it trains on the CPU.

Fixes SMF-1055

Changes

tutorials/gpu-node-pools/, in the shape of the other tutorials, for the cluster that scripts/01-user-assigned-managed-identity.sh creates:

• 01-add-gpu-node-pool.sh adds gpupool: Spot Standard_NV18ads_A10_v5 VMs (eviction policy Delete, maximum price -1) on Azure Linux, the cluster autoscaler from 0 to 1 node, the label workload=gpu and the taint sku=gpu:NoSchedule. It enables the KEDA add-on if the cluster lacks it, and stops when a pool of that name exists with another VM size or priority, since neither can change.
• 02-deploy-device-plugin.sh deploys the NVIDIA device plugin DaemonSet of Microsoft's manual install guide (nvcr.io/nvidia/k8s-device-plugin:v0.18.0), plus a node selector for the accelerator=nvidia label AKS gives GPU nodes and a toleration for the Spot taint.
• 03-create-resources.sh creates the Service Bus namespace and the training-requests queue, and the managed identity the KEDA tutorials share (local-keda-uami-test) unless one of them already did. The identity gets Azure Service Bus Data Owner on the namespace, granted on its principal id, and is federated with the KEDA operator and with the job's service account. The operator is bound to it, and restarted only when the binding changes.
• 04-deploy-scaled-job.sh deploys the job's service account, the training script as a config map, the TriggerAuthentication and the ScaledJob: one job per message, whose pod asks for nvidia.com/gpu: 1, selects the pool's nodes and tolerates its taints. It derives the Service Bus endpoints when it applies them, so the same manifests serve both targets: on the emulator the job connects through a custom endpoint address, which makes the Python SDK use AMQP over WebSockets and so needs websocket-client, and the scaler reads the HTTPS API on the gateway port.
• 05-run-training-job.sh sends a training request and follows its job: the autoscaler adds a GPU node, and the script prints the node's labels, taints and GPUs, then the job's log.
• 06-simulate-eviction.sh sends a 10-minute training request, waits until the job's pod logs training progress, runs az vmss simulate-eviction on that pod's node, follows the node until AKS deletes it, and waits for the job to complete on a new node.
• 07-cleanup.sh deletes what the tutorial created, and keeps the identity the KEDA tutorials share.
• src/train.py receives one message with DefaultAzureCredential, which signs in with the pod's workload identity, and trains a small PyTorch model on cuda when PyTorch finds a GPU, on cpu otherwise.
• The tutorial's README.md, and its row in the root README.md.

The scripts follow the repository's conventions (#!/bin/bash, 00-variables.sh, manifests filled in with yq, a check after every command that can fail) and need nothing beyond bash 3.2.

Tests

• Static: bash -n on every script with bash 5 and with GNU bash 3.2.57, the version macOS ships; shfmt -i 2 -d; src/train.py parses; the Mermaid diagram validates; the repository's pre-commit hooks (gitleaks passed, zizmor has no workflow to check).
• LocalStack for Azure, the published image, on a fresh emulator each time, routed with lstk az start-interception. The emulator ran with docker run as the samples workflow's Windows job starts it, with the external service ports from 4512, because another program held port 4511 on the test machine and lstk start checks the whole range. scripts/01-user-assigned-managed-identity.sh created the cluster, then the scripts ran in order, with 01 to 03 run twice: every step exited 0, and the second runs changed nothing.
  • On the head (ae8665e), with latest as published on 2026-11-05 (sha256:b728ec0a...): the cluster in 385 s. 05: the cluster autoscaler added a GPU node, and the job received its message and trained on cpu (CUDA available: False). 06: the script waited about 50 s for the pod's training progress, AKS tainted the node remediator.kubernetes.azure.com/unschedulable:NoSchedule and deleted it 42 s after the eviction request, and the replacement pod found the queue empty and completed on a new node. 07 deleted everything it lists.
  • On the first commit, with latest as published on 2026-11-04 at 14:42 UTC (sha256:31573c66..., the first latest with GPU node pool support), every step passed as well, and so did an earlier run on the emulator built from source with the same AKS code.
• Azure, italynorth, three runs, the last on the head. The cluster came from az aks create with --enable-keda --enable-oidc-issuer --enable-workload-identity and the names of the repository's script, because scripts/01-user-assigned-managed-identity.sh needs the object id of a Microsoft Entra admin group, which the test service principal can't look up. Every step exited 0, and the second runs of 01 to 03 changed nothing.
  • 05: the cluster autoscaler added a Spot Standard_NV18ads_A10_v5 node, the device plugin advertised one nvidia.com/gpu, and the job trained on cuda on an NVIDIA A10-12Q, 250 s in all.
  • 06: the script waited about 40 s for the pod's training progress, AKS added the same eviction taint and deleted the node 425 s after the eviction request (154 and 158 s in the two earlier runs), and the replacement pod found the queue empty and completed on a new node, on cuda.
  • The node used 18 of the subscription's 32 Spot vCPUs in the region, while the Standard NVADSA10v5 Family quota is 0, which Spot doesn't draw on. 07 and the deletion of the resource group left no resource behind and no Spot vCPU in use.
• Parity: the pool 01 prints, the node 05 prints (labels, taints and GPUs) and the eviction taints are identical on the emulator and on Azure.
• Not run: on macOS beyond the bash 3.2 parse, from Git Bash on Windows, and on Azure with the repository's cluster script.

🤖 Generated with Claude Code